Renegade
Play as either a student or detective and solve crimes around the open world.
Team: Group Project (2 Members)
Role: Game & Level Designer, Programmer
Timeline: 8 Weeks
Tools Used: Unreal Engine, Adobe Photoshop, Mixamo, Sketchfab, Fuse
Platform: PC
Project Accomplishments
1. The ChatGPT technology allows the player to ask the NPC anything, and the AI will react with a textual response that is then converted to speech via the Azure speech integration.
2. ChatGPT has been integrated into the NPCs, allowing them to have conversations with each other in the open world. The player can roam around town and actually hear people talking to one another, making it feel like a real-world experience.
3. We have also given the player the option to convert his speech into text to interact with the AI through Azure Speech.
2. ChatGPT has been integrated into the NPCs, allowing them to have conversations with each other in the open world. The player can roam around town and actually hear people talking to one another, making it feel like a real-world experience.
3. We have also given the player the option to convert his speech into text to interact with the AI through Azure Speech.
Gameplay
Level Design
Core Design Challenges
Some of the challenges that we faced during the development of this project are:
1. Since the ChatGPT API lacks memory, it was not taking into account the user's previous cues and responses when triggering a conversation.
2. There is not a lot of information available regarding Unreal Engine 5.1 because it is so new. Similarly, we struggled to understand how to include Azure APIs with our NPCs.
2. There is not a lot of information available regarding Unreal Engine 5.1 because it is so new. Similarly, we struggled to understand how to include Azure APIs with our NPCs.
How did I tackle these challenges?
1. To get around the lack of memory, whenever we send a prompt to an NPC, we include their most recent response from ChatGPT.
2. We tried to find unreal engine documentation and tutorials online for the most helpful topics.
Top-Down Map
Environment Design
Game Design
Core Mechanics
1. Player Controller
The player controller includes the following features:
1. Movement around the map
2. Jumping over obstacles / taking cover behind obstacles
3. Ability to switch between first-person and third-person camera
4. Talk to NPCs through text or speech
5. Drive Cars
1. Movement around the map
2. Jumping over obstacles / taking cover behind obstacles
3. Ability to switch between first-person and third-person camera
4. Talk to NPCs through text or speech
5. Drive Cars
2. AI/NPC
The NPC AI has been custom-made by us using the Fuse software and includes the following features:
1. Navigation around the map to random locations
2. Player Interaction
3. Ability to follow player
4. Voiced Dialogues
1. Navigation around the map to random locations
2. Player Interaction
3. Ability to follow player
4. Voiced Dialogues
3. Azure Text-to-Speech Implementation
The text entered by the user in the text box is converted to a sound wave using the Azure Speech service. Then, a new audio source is spawned on the main character to play the generated sound, while at the same time, the main character is animated while speaking, allowing us to provide players with a lifelike conversation experience. After the character has spoken the converted speech, it is sent as a prompt to ChatGPT, as described below.
4. Azure Speech-to-Text Implementation
Audio input from the player's microphone is converted to text, allowing us to use it in the prompt sent to ChatGPT for NPC responses, as well as to make the main character utter the extracted text using Text to Speech, as described above. The player can interact with NPCs through both textual and audible input, which is practical for players who favour either method. It is also possible to disable the main character's animation when using audio input to interact with non-player characters. We are sending the previous response in the prompt because the ChatGPT API doesn’t have memory and can’t remember previous responses without it.
Outcomes
What did I learn?
1. Creating a third-person game in Unreal is not as easy as in Unity; I had to learn all about Unreal Engine 5 throughout the process.
2. Using ChatGPT to make AI in games.
3. Using Azure AI for speech generation.
